A variety of input or control devices have been developed for interaction with microprocessor- or computer-based electronic devices. For example, keyboards have been developed to enter text-based data, which can include commands, to an electronic device. Buttons and switches have also been developed to allow a user to indicate or select a desired operating condition or other control parameter. Graphical user interfaces (GUI) have also been developed to provide a user friendly and intuitive interface for providing control inputs. A GUI system typically includes a graphical display that portrays one or more control icons and a user moveable and selectable cursor that is moved about the display. Via the moveable cursor, a user can select or indicate a control icon and select a desired control action.
Current approaches for mode switching between basic interaction tasks are device-specific and quite different from the direct manipulation style that most users are accustomed to. More importantly, some of these approaches cannot be employed in pen-based interaction because they require multiple contact points (e.g., multi-touch and multi-finger commands for zooming).
Pen-based devices are gaining popularity. Smart phones, tablet computers and digitizing tablet displays have penetrated our lives to a great extent due to their mobility, ease of use and affordable prices. However, despite what their name suggests, pen-based devices are not purely pen-based. For example, in pen-based smart phones, many actions force the user to put the pen aside and switch to multi-finger gestures (e.g. spread/pinch for zoom in/out, and swipe to navigate back/forward). These gestures require the simultaneous use of 2, 3 or even 4 fingers. The necessity of switching between pen and multi-touch input goes against the goal of seamless interaction in pen-based devices. Another example where we lose purely pen-based interaction is with tablet computers. In most pen-based applications, features are hidden in standard context/pop-up menus that are accessed via tapping and/or holding the pen on the tablet screen in various ways. In this case, the pen is used to emulate a mouse, which fits the traditional GUI/WIMP-based interaction paradigm, rather than that of a purely pen-based interaction. Even the state of the art devices and software specifically built for pen-based interaction lack purely pen-based interaction. For example, graphics tablets preferred mainly by digital artists such as Wacom Cintiq 24HD are often referred to as “heaven on earth” by users. However, even with these high-end models many tasks are still accomplished via on-pen or on-tablet external buttons called “express keys”, “touch rings” and “radial menus”. These buttons allow the user to simulate keystrokes including letters, numbers and modifier keys (e.g. Shift, Alt and Control). To issue a virtual manipulation command (e.g. scroll), the user has to locate the correct button which interrupts the interaction flow, hence causing an overall disappointing experience. In addition to not being purely pen-based, the use of smart gestures, soft menus, and external buttons has an adverse effect on adaptation. For example, as the number of tasks increases so does the diversity of smart gestures, available menus and external buttons that a user must learn and get accustomed to. Moreover, these methods of interaction are brand-, device- and application-specific, which, in fact, makes the situation even more complicated for users. The users have to dedicate a considerable amount of time and effort before they fully discover and start using various functionalities offered by their pen-based devices. However, users frequently have little interest in allocating time and effort for instruction and they would rather take a “walk-up-and-use” approach to software interfaces. These issues show that existing pen-based systems depend substantially on multi-finger gestures, context/pop-up menus and external buttons which goes against the philosophy of pen-based interfaces as a more intuitive interaction alternative.
In the embodiment of the prior art, command interfaces are used. The command interfaces are based on the eye-mind hypothesis M which intentional eye movements are associated with interface actions. In other words, in command interfaces, gaze is employed as an explicit computer peripheral device. This embodiment requires the gaze to be used for manipulation in addition to its natural purpose, visual perception.
In addition, this embodiment forces the user to be aware of the role of the eye gaze and therefore causes high cognitive workload.
In another embodiment of the prior art, non-command interfaces are used. The non-command interfaces are based on the computer system passively and continuously observing the user in real-time and providing appropriate responses. In order to provide satisfying and natural responses, the computer system must be able to infer user's intentions from his/her spontaneous natural behaviors. An intention can be, for instance, moving a window, scrolling a piece of text or maximizing an image. However, the majority of the related work on non-command interfaces focuses solely on monitoring and post-hoc analysis of eye movements collected dimming natural interaction.
U.S. Pat. No. 6,873,314 discloses a method and system for the recognition of reading, skimming and scanning from eye-gaze patterns. Eye-gaze patterns are accurately recognized when a user is reading, skimming, or scanning on a display filled with heterogeneous content, and then the method and system supply information tailored to meet individual needs. Heterogeneous content includes objects normally encountered on computer monitors, such as text, images, hyperlinks, windows, icons, and menus. Three distinct mechanisms are used; coarse or quantized representation of eye-movements, accumulation of pooled numerical evidence based detection, and mode switching. Analysis of text the user is reading or skimming may infer user interest and adapt to the user's needs.
GUI systems have employed a variety of interfaces for interacting with the GUI system to move and select the cursor as displayed on the display screen. A mouse can have a wired or wireless connection to the electronic device. Physical movement of the mouse, for example on a table, desk, mouse pad, and the like, are translated into corresponding movements of the cursor. A button or pressure switch is typically provided to allow a user to activate or “click” the cursor when moved to the desired location. Trackballs work in a similar manner; however movement of the cursor is induced via rotational movement of the trackball rather than the gross movement of a mouse. Track pads or mouse pads allow a user to trace or swipe their fingertip across the pads to effect movement of the cursor. Tapping on the track pad or activation of a separate switch activates or clicks the cursor. Electronic devices can also be provided with touch screens such that a user may indicate directly on a display the desired control icon and corresponding control input.
These computing devices typically require a separate control device, such as a mouse or game controller, to interact with the computing device's user interface. Users typically use a cursor or other selection tool displayed in the user interface to select objects by pushing buttons on the control device. Users also use the control device to modify and control those selected objects (e.g., by pressing additional buttons on the control device or moving the control device). Training is usually required to teach the user how movements of this control device map to the remote user interface objects. Even after the training, the user sometimes still finds the movements to be awkward.